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ABSTRACT 

Two yeare ago at thie conference, Dr. 
Sherrie Gott of the Air Force Human 
Reeourcee Laboratory deecribed an avio- 
nice troubleehoo t ing tutor being deve- 
loped under the the Baeic Job Ski lie 
Reeearch Program. The tutor, known ae 
Sherlock, ie directed at teaching the 
diagnoetic proceduree neceeeary to inves- 
tlgate complex teet equipment ueed to 
maintain F-13 fighter aircraft. Since 
Dr. Gott'S preeentation in 1087, the 
tutor hae undergone field teeting at two 
Air Force F-15 flying winge. The reeulte 
of the field teet ehowed that after an 
average of 20 houre on the tutor, the 15 
airmen in the experimental group (who 
averaged 28 monthe of experience) ehowed 
eignificant performance gaine when com- 
pared to a control group (having a mean 
experience level of 37 monthe) who conti- 
nued participating in the exieting on- 
the-job training program. Troubleshoot- 
ing performance of the tutored group 
approached the level of proficiency of 
highly experienced airmen (averaging 
approximately 114 monthe of experience), 
and theee performance gaine were 
confirmed in delayed teeting eix monthe 
following the intervention. The tutor ie 
currently undergoing a hardware and 
eoftware convereion from a Xerox Lisp 
environment to a PC-baaed environment 
ueing an object-oriented programming lan- 
guage. Thie paper summarizes the reeulte 
of the eucceeeful field teet and focueaee 
on (a) the ine truct ional featuree that 
contributed to Sherlock's success, and 
(b) the implementation of theee features 
in the PC-baeed version of the avionics 
troubleshooting tutor. 

INTRODUCTION 

In developing the avionics troubleshoot- 
ing tutor to be deecribed in this paper, 
the Basic Job Skills Program attempted to 
address several fundamental problems that 
the Air Force maintenance community faces 
with respect to the training of mainten- 


ance technicians. First, while the com- 
plexity of the systems to be maintained 
is increasing with advances in aerospace 
technology, there hae been no correspon- 
ding increase in the time available to 
new trainees to learn about these sys- 
tems . As a result, the time needed to 
acquire the knowledge necessary to per- 
form these Jobs increases and the Air 
Force has received fewer of the benefits 
of its training by the time maintenance 
personnel leave the Air Force. 

One response to the increase in the tech- 
nical complexity of these jobs has been 
to provide technicicans with procadura- 
lized Job aids and so-called "smart" 
machines equipped with sel f -diagnostic 
capabilities. The rationale supporting 
this response is that in providing tech- 
nicians with cook-book procedures for 
dealing with maintenance problems they 
might encounter, together with machines 
that diagnose their own faults, one can 
reduce the knowledge and skill required 
of the human technician and still main- 
tain the productivity of the work force. 
Unfortunately, the adequacy of procedura- 
lized Job aids is limited by the fact 
that, given the complexity of current 
aircraft systems , even the best designers 
are unable to anticipate every conceiv- 
able fault or fault combination that 
their system might develop. There are 
also limits to the diagnostic capabili- 
ties of automated systems. For example, 
Gott ( 1087) cited a 65% hit rate for the 
diagnostics of some systems on the BIB. 
Thus, there Is still a clear need for 
human expertise to pick up where procedu- 
ral aids and automated diagnostics leave 
off. A related consequence is that re- 
liance on Such aids gives technicians a 
false sense of security and undermines 
the development of the expertise that 
will invariably be required when the 
technician is confronted with a novel set 
of conditions for which the proper repair 
procedures have not been prespecified. 

A final dimension of the maintenance 
training has to do with the fact that the 
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first priority in the shops where techni- 
cian# receive their on-tha-Job training 
i« raraly to train new 'technician* , but 
to kaap plana# flying. Thi# mean# tur- 
ning out unit# that coma into the ahop 
for repair a# quickly a# po##ibie. In 
order to meat thi# goal and to kaap inex- 
perienced personnel from breaking expen- 
sive equipment, it i# often only the most 
experienced technician# who work on the 
more difficult problem#. Thu#, trainee# 
are denied important learning opportuni- 
ties where practice at solving difficult 
diagnostic problem# would promote their 
understanding of the task and tEe system 
they are working with. 

DEVELOPMENT AND EVALUATION OF AN AVIONICS 
TROUBLESHOOTING TUTOR 

In order to ameliorate these effect#, the 
avionic# troubleshooting tutor, Sherlock, 
was designed to provide trainee# with the 
type of troubleshooting practice that 
would decrease both the total reliance of 
novice technicians on automated diagnos- 
tics and procedural ised Job aids, a# well 
as the amount of time required to acheive 
proficiency in the ta«k of maintaining 
aircraft systems. The design was based 
on analyses of expert troubleshooting 
performance (Gitomer, 1084, 1088; Glaser 
et al., 1083; Gott, Bennett, and Qillett, 
1086) which identified three cognitive 
components of their expertise: the know- 

ledge underlying experts 1 use of trouble- 
shooting procedures such as tracing elec- 
trical signals using schematics and 
taking measurements of the signals; the 
strategic knowledge underlying decisions 
regarding appropriate actions to take 
given multiple alternatives; and the 
declarative knowledge of the system it- 
self which allows experts to accurately 
represent the problem and thereby con- 
struct and constrain the problem space . 
Sherlock incorporates a series of 34 
troubleshooting scenarios that are 
designed to foster these multiple types 
of expert knowledge. The scenarios are 
presented to students in an ordered se- 
quence. This sequence was informed by 
the examination of novice weaknesses in 
the cognitive task analysis, and was 
designed to foster increasingly sophisti- 
cated models of the test equipment and 
the troubleshooting task. 

Sherlock was evaluated in a controlled 
experiment at two Air Force F-13 flying 
wings (Nichols, Pokorny, Jones, Gott, and 
Alley, in preparation; Gott, 1080). A 
verbal troubleshooting test was used to 
identify 32 avionics technicians who had 
either beginning or intermediate trouble- 
shooting skills (see Nichols, et al . for 
a complete description of the verbal 
troubleshooting task). On the basis of 
their performance, subjects were ranked 
within testing site and matched pairs 


were established. One member of each 
matched pair was then randomly assigned 
to either the experimental or control 
group such that half the subjects at each 
testing site were assigned to each group. 
Subjects' scores on the verbal trouble- 
shooting task provided a baseline measure 
against which performance gains could be 
measured post-exper imental ly . The pre- 
test scores revealed no significant dif- 
ferences between groups in performance on 
either the verbal troubleshooting prob- 
lems that were administered at both bases 
or on a number of other indicators that 
were used to corroborate the equality of 
groups prior to the intervention (see 
Nichols, et al . for a complete descrip- 
tion of these measures) . 

The experimental subjects received an 
average of 20 hours on Sherlock over the 
course of approximately three weeks while 
the control subjects continued their on- 
the-job training. Parallel forms of 
pretest measures were then readministered 
as posttests by researchers who were 
blind with respect to individual sub- 
jects participation in either the expe- 
rimental or control group. Figure 1 
shows differences in pre- and post -test 
performance on "the “verbal troubleshooting 
task for the two groups. An independent 
samples t-test revealed no significant 
differences between mean pre-test scores 
of 53.40 for the control group and 56.03 
for the tutor group (t(30)=0.38, p>.5, 
two tailed). Post-test performance, how- 
ever, differed significantly (F(l,29)» 
15.62, p< . 01 ) , with tutored subjects 
obtaining a mean score approximately 20 
points higher than that of control sub- 
jects. In order to get some idea of what 
this performance gain translates to in 
terms of increased experience, a group of 
skilled airmen with an average of 114 
months of experience in this career field 
was tested on the verbal troubleshooting 
task. Their mean score is plotted in the 
upper left-hand corner of Figure 1, and 
is quite similar to that of the tutored 
group who had an average of only 28 
months of experience. When experimental 
and control subjects were retested 5 to 6 
months after the experiment had been - 
conducted, the tutor's effect persisted 
with tutored subjects achieving a mean 
score approximately 15 points higher than 
that of the control group. Wien compared 
with their immediate posttest perfor- 
mance, the slight performance decrement 
of the tutored group on the delayed post- 
test was not s tatistical ly significant. 

The success of the Sherlock field test 
has resulted in high-level support for 
the BJS program from within Tactical Air 
Command which employs the maintenance 
personnel whose training Sherlock tar- 
gets. In order to get the tutor into 
Air Force maintenance work places, Sher- 
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lock is currently undergoing a hardware 
and software conversion which will allow 
the system to be delivered on standard PC 
hardware that is available in maintenance 
work centers. This conversion is being 
carried out at the University of Pitts- 
burgh Learning Research and Development 
Center by researchers responsible for the 
original development of Sherlock. In 
addition to the need for delivering Sher- 
lock on standard Air Force hardware, 
decisions regarding Sherlock's conversion 
have been driven by three primary con- 
cerns: first, the instructional features 

that led to the tutor's success must be 
better understood and retained; secondly, 
the tutor's limitations must be expli- 
cated and reduced; and finally, the 
resultant tutor courseware must be main- 
tainable by Air Force personnel. 

INSTRUCTIONAL FEATURES OF THE AVIONICS 
TROUBLESHOOTING TUTOR: TROUBLESHOOTING 

PRACTICE IN A SIMULATED, SUPPORTED WORK 
ENVIORNMENT 

The instructional features of Sherlock 
that appear to be responsible for the 
dramatic learning gains are associated 
with the simulated, supported practice 
environment that the tutor provides. 
Specifically, opportunities for realistic 
practice, feedback to foster the develop- 
ment of a mental model of an electronic 
teat, menus that support the development 
of goal-oriented activity, and multiple 
levels of hints from Sherlock’s coach are 
of particular Interest. 

One of Sherlock’s moat important instruc- 
tional features is that it provides stu- 
dents with the opportunity to practice 
solving realistic troubleshooting prob- 
lems in a simulated but supported work 
environment. Figure 2 shows the tutor 
display as it appears to the student upon 
presentation of a troubleshooting prob- 
lem. The context of the problem is 
established by presentation to the stu- 
dent of a scenario that technicians might 
encounter on the job. The problem is 
thus presented in much the same way that 
a real problem would present itself in 
the shop. The work environment of the 
shop is also represented in the form of a 
simulated test station, a unit from the 
Jet that is being tested (referred to as 
a line replaceable unit or LRU) , and a 
test package connecting the LRU to the 
test station. The simulated dimensions 
of the equipment are primarily the exter- 
nal controls of test station drawers 
rather than their internal functional 
behavior. Front panels of test station 
drawers were graphically simulated to 
appear as similar to the real work envi- 
ronment as possible, and indicators and 
controls were functionally simulated to 
allow manipulation by the student for the 
purpose of performing tests and taking 


measurements. Within the test station, 
measurements are taken by selecting test 
points on schematic diagrams displayed on 
Sherlock's screen. Measurement values 
have been prespecified, however , and do 
not result from an underlying deep simu- 
lation of the device (i.e., test equip- 
ment and LRU) . 

In most of Sherlock's problems, as in the 
real shop environment, a corrective ac- 
tion or 'fix" called out by the technical 
orders for a failed test step rarely 
fixes the problem. It is at this point 
then that students must begin to think on 
their own to develop a plan for isolating 
the fault. This requires relating the 
failed test to a mental model of the 
system as it was presumed to be functio- 
ning at the time of the fail. This envi- 
sioning process involves representing 
components of the system that were active 
during the test, and the flow of informa- 
tion through these components. Figure 3 
illustrates an abstract model of an elec- 
tronic test which can be used to charac- 
terize any circuit path that the student 
might have to investigate. A stimulus 
signal is generated by one of the drawers 
in the test station, and sent to a rou- 
ting device which routes the signal 
through the test package and the LRU. The 
LRU responds to the input signal and 
produces an output which is sent back to 
the test station and routed to a measure- 
ment device (Lesgold, Lajoie, Bunzo, and 
Eggan, 1988). In relating this abstrac- 
tion to a particular test, the student is 
encouraged to identify the active circuit 
path for that test. The model of the test 
thus provides a structure for the organi- 
zation of the student's declarative know- 
ledge of the system and constrains the 
search for the fault. 

The tutor is also directed at the deve 
lopment of goal structures for investiga- 
ting the equipment, procedural knowledge 
of specific troubleshooting actions, and 
additional strategic knowledge required 
to inform decision making during problem 
solving (Gott, 1989). Sherlock's action 
menu, shown at the right-hand side of the 
display reproduced in Figure 2, allows 
students to choose which area of the 
equipment they want to investigate, and 
to select the procedures for doing so. 
Some of these menu selections have addi- 
tional choices embedded within them 
representing further decisions that the 
technician must make in pursuing a parti- 
cular solution path. The menus serve to 
structure the problem-solving process and 
facilitate the apprentice's development 
of a conceptual model of the task. Thus, 
for example, in testing an LRU that has 
come in from the flight line, the student 
must access the technical order that 
describes the test procedures for that 
particular LRU, set up the drawers as 
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instructed for each test on the LRU 
(e.g. f wiring integrity tests, power 
short tests, resistance tests, etc.), and 
run and interpret each test. If a test 
fails, the technical order might call out 
a suggested fix for the fault, and the 
student Is encouraged to try that fix 
before investigating other components as 
the cause of the failed test. Other 
procedural choices represented in Sher- 
lock's action menus include selection of 
test points, selection of components to 
be replaced, swapping suspected bad com- 
ponents, checking connections, etc. 
Sherlock thereby provides a simulated 
learning and practice environment so that 
technicians can exercise the skills they 
must use in the real work environment. 
Moreover, Sherlock embodies a coach or 
master technician to foster apprentice- 
ship learning with feedback and general 
problem solving assistance. 

Sherlock's coach offers external support 
In the form of hints that are provided 
when the student asks for help. The 
hints, like the action menu choices, are 
tied to the goal structure of fault iso- 
lation tasks, and vary according to type 
and level of expl ici tness . Hint type is 
related to the student's current trouble- 
shooting activity and specifies, for 
example, where to take a measurement or 
how to interpret a measurement already 
taken. The explicitness of the hint is 
determined by the student who can access 
up to five levels of increasingly direc- 
tive hints, from a simple recap of past 
plans and actions, to detailed informa- 
tion concerning how to perform the next 
suggested action. Unsolicited interven- 
tion from the coach can also occur under 
certain circumstances, for example, If 
the student fails to turn off a hazardous 
voltage prior to extending a circuit 
card, or investigates a piece of equip- 
ment that was not being used when the 
test failed. Sherlock's hints are thus 
adaptive in the sense that hints received 
are dependent on the individual student's 
activity at the time the hint is ac- 
cessed, and the desired level of assist- 
ance as indicated by the specificity of 
the hint requested. 

Sherlock's ins true t ional limitations re- 
sult primarily from the fact that the 
tutor's curriculum is to a large extent 
prespecified. The problems presented to 
students and their sequence is the same 
for all students, regardless of their 
individual strengths and weaknesses. Al- 
though the tutor evaluates students' 
problem solutions and highlights their 
strengths and weaknesses in post-problem 
feedback, this diagnostic capability is 
not exploited to provide problems that 
are particularly adapted to the indivi- 
dual student’s current level of skill. 

This lack of adaptiveness exists because 


the tutor does not possess the capability 
of generating new problems on line in the 
course of tutoring. Further, on-line 
diagnosis of students' troubleshooting is 
not robust enough to determine the appro- 
priate type and level of hint to provide 
when a student asks for help. While the 
presentation of hints is adaptive in the 
sense described earlier, the hints them- 
selves have been prespecified and the 
principles that determine hint content 
and guide Sherlock’s decisions to inter- 
vene are not as yet clearly established. 
In the next version of the tutor, simula- 
tion will be deeper in the sense that a 
set of circuits will be functionally 
simulated and the electronic tests per- 
formed on these circuits will be mo- 
delled. This simulation will provide the 
basis for improved student modelling and 
diagnosis, on-line problem generation, 
and more principled explanations and 
student feedback (see following section 
for a more complete discussion of how 
these improvements will be implemented) . 

THE AVIONICS TROUBLESHOOTING TUTOR II 

The next generation of Sherlock is pre- 
sently under development, with the con- 
cerns described above providing the foci 
for the effort. The goals include deli- 
very of instruction on accessible, cost- 
effective hardware, simplification of 
tutor development and maintenance by Air 
Force personnel, and increased adaptive- 
ness in instruction, including improved 
student diagnosis and on-line problem 
generation capability. 

Sherlock was originally implemented in 
the Xerox Lisp environment in order to 
take advantage of its large internal 
memory capacity and superior graphics 
capabilities . The “idea - Was to first test 
the validity of the cognitive models and 
theory underlying Sherlock’s design uti- 
lizing optimal computer hardware. Now 
that the theoretical and empirical bases 
of Sherlock have been tested and suppor- 
ted, we must consider ways of delivering, 
the tutor on a scaled-down system without 
sacrificing essential performance charac- 
teristics of more powerful machines. The 
basic configuration of the Avionics 
Troubleshooting Tutor II is depicted in 
Figure 4. The system consists of an 
80386-based PC with one MB of Internal 
memory and two to three MB of expansion 
memory. The PC is connected, via an RS- 
232 cable, to a video disc player which 
stores video images to be displayed on a 
20-inch multiscan, high-resolut ion moni- 
tor. The PC is equipped with VGA graphics 
and a super lmposer board for overlaying 
computer graphics on video images. 

The use of video in displaying the work 
environment provides several advantages 
over computer graphics. First, by using 
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video imaged of the real tea t equipment , 
Sherlock ' s feature of providing a realis- 
tic work environment id retained, and in 
fact, enhanced with concrete visual rep- 
resentations of actual physical equip- 
ment. Figure 5 shows the front panel of 
one of the test station drawers in the 
original Sherlock. The time required to 
develop such detailed graphics and all 
possible configurations of each front 
panel, not to mention their storage re- 
quirements, represents a significant in- 
vestment of resources to achieve work 
environment realism in Sherlock I. That 
investment will be significantly reduced 
via the use of video in Sherlock II. In 
that version of the tutor, computer 
graphics will be used almost exclusively 
in the menus, with a resultant savings in 
development time and storage. Second, 
because the Air Force maintenance commu- 
nity currently uses interactive video in 
developing (and delivering) its mainten- 
ance training, instructional designers 
who will ultimately maintain the tutor 
are already familiar with the technology. 

The converted tutor is being developed in 
the Smalltalk V286 software environment 
which will allow significant savings in 
development time and facilitate the im- 
plementation of more adaptive instruction 
in the form of on-line hint and problem 
generation. Recall that the converted 
tutor will employ simulated circuits. 

The object-oriented environment provided 
by Smalltalk was chosen since it Imple- 
ments sophisticated class structures with 
asynchronous messaging between objects, 
thereby allowing for the simulation of 
complex systems. The environment also 
reduces development time because the 
object class structure defined by the 
programmer determines the properties of 
objects within a class and the operations 
that can be applied to them. Put another 
way, objects inherit the properties and 
operations (or methods) of their class 
which are defined only once for the en- 
tire class rather than for each object 
within the class. The reduction in deve- 
lopment time thus results from the abili- 
ty to, in effect, use a given piece of 
code for multiple purposes. 

The implementation of a class structure 
is illustrated by the simulated circuits 
being developed for Sherlock II, and the 
electronic tests which operate on the 
circuit. Each instance of a test in- 
volves four elements: a signal source, an 
LRU (the unit being tested) , a measure- 
ment device, and a circuit path. Al- 
though different tests may use different 
circuit paths, stimulus sources, etc. , 
all instances of each element have cer- 
tain behaviors in common. For instance, 
a broken wire in any circuit path will, 
in general , cause an ohms measurement to 
indicate infinite resistance. If, in a 


particular circuit, the wire was shorted 
to another wire, the reading might be 
different. It is only under these un- 
usual types of conditions that additional 
code must be written to override the 
behavior that defines circuits in gene- 
ral. Thus, rather than coding each cir- 
cuit to be used in the tests independent- 
ly, all circuits will share code that 
defines their common behaviors. The 
ability to capture the general properties 
of objects will provide the basis for 
rule-based problem generation and hint 
generation In Sherlock II. Given that 
problems and hints can be created by 
rule, then they can be generated during 
the course of the tutoring session in a 
way that is responsive to the indivi- 
dual’s troubleshooting strengths and 
weaknesses , thus providing more adaptive 
instruction . 

The object-oriented programming environ- 
ment also promotes maintainability of the 
tutor because it lends itself to modula- 
rization. Modular code makes the struc- 
ture of the program clearer, thus facili- 
tating modifications by programmers who 
were not involved in the tutor's original 
development. Modules that do not require 
modification for new versions of the 
tutor or for other tutors being developed 
for different maintenance Job specialties 
can also be easily transported. The BJS 
Program is currently conducting a trai- 
ning needs assessment to determine the 
type of programming expertise required 
for maintenance of intelligent tutors 
developed In the object-oriented lan- 
guages so that this task can be performed 
by Air Force personnel. 

CONCLUSIONS 

In addressing the needs of the Air Force 
maintenance community, the Basic Job 
Skills Program has benefitted from metho- 
dological and theoretical advances in 
cognitive science. These benefits are 
reflected in, for instance, the cognitive 
task analysis procedure which was used to 
inform Sherlock’s design (Gott, 1087) and 
the increasingly comprehensive models of 
troubleshooting performance that the re- 
sults of these analyses yield. To the 
extent that the cognitive approacn 
Sherlock’s design contributed to the 
tutor's effectiveness, then an important 
future goal for the BJS Program will be 
to make this technology available to 
nonscientists in the Air Force who are 
responsible for instructional design and 
maintenance of educational courseware. 
Steps toward this goal include the deve- 
lopment of maintainable software that is 
compatible with standard hardware, and 
the attempt to gain a better understan- 
ding of the instructional approach re- 
quired in teaching a complex skill like 
troubleshooting . 
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FIG. 1. RESULTS OF FIELD TEST 
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FIG. 2. WORK ENVIRONMENT DISPLAY 
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FIG. 3. AVIONICS EQUIPMENT CONFIGURATION 

(SIGNAL PATH) 
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FIG. 4. CONFIGURATION OF 
AVIONICS TROUBLESHOOTING TUTOR II 
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FIG. 5. FRONT PANEL DISPLAY OF A 
TEST STATION DRAWER IN SHERLOCK 








